Uniformization in Markov Decision Processes

Continuous-time Markov decision processes (CTMDP) may be viewed as a specialcase of semi-Markov decision processes (SMDP) where the intertransition times are exponen-tially distributed and the decision maker is allowed to choose actions whenever the systemstate changes. When the transition rates are identical for each state and action pair, one canconvert a CTMDP into an equivalent discrete-time Markov decision process (DTMDP), whichis easier to analyze and solve. In this article, we describe uniformization that uses fictitioustransitions from a state to itself and hence enables the conversion of a CTMDP with nonidenti-cal transition rates into an equivalent DTMDP. We first demonstrate the use of uniformizationin converting a continuous-time Markov chain into an equivalent discrete-time Markov chain,and then describe how it is used in the context of CTMDPs with discounted reward criterion.We also present examples for the use of uniformization in continuous-time Markov models.